A Distributed JAVA SPACE Implementation for HARNESS

Distributed Virtual Machine (DVM) Heterogeneous computational resources Services (plug-ins) Baseline Figure 1: Overview of the Harness system. 2.1 Fundamental Abstractions and System Architecture The fundamental abstraction in the Harness metacomputing framework is the Distributed Virtual Machine (DVM) (see gure 1, level 1). A DVM is associated with a symbolic name that is unique within a Harness name space, but has no physical entities connected to it. Heterogeneous Computational Resources may enroll into a DVM (see gure 1, level 2) at any time; however at this level the DVM is not ready yet for use by applications. To adapt to applications' needs, the heterogeneous computational resources enrolled in a DVM need to load plug-ins (see gure 1, level 3). A plug-in is a software component implementing a speci c service. By loading plug-ins that implement services, it is possible to complement the set of native services of a computational resource in such a way that all the computational resources enrolled in a DVM present a consistently homogeneous service baseline to applications (see gure 1, level 4, Baseline). Users may recon gure the DVM at any time (see gure 1, level 4) both in terms of computational resources enrolled by having them join or leave the DVM, and in terms of services available by loading and unloading plug-ins. The main goal of the Harness metacomputing framework is to support the ability to enroll heterogeneous computational resources into a DVM and make them capable of delivering a consistent service baseline to users. This goal requires the programs comprising the framework to be as portable as possible over as large as possible a selection of systems. The availability of services to heterogeneous computational resources derives from two di erent properties of the framework: the portability of plug-ins and the presence of multiple searchable plug-in repositories. Harness implements these properties primarily by leveraging two di erent features of Java technology. These features are the capability to layer a homogeneous architecture such as the Java Virtual Machine (JVM) [14] over a large set of heterogeneous computational resources, and the capability to customize the mechanism provided to load and link new objects and libraries. However, the adoption of the Java language as the development platform for the Harness metacomputing framework has given us several other advantages: The Harness framework is realized as a collection of cooperating objects with consistent 3 boundaries (Java classes) that present a coherent and stable object-oriented development environment to users. A clear and consistent boundary is de ned for plug-ins, in fact each plug-in is required to appear to the system as a Java class. All the entities in the framework could be implemented using a robust multi-threaded architecture. Users can develop additional services both in a passive, library-like avor and in an active thread-enabled avor. A simple Object Oriented mechanism is available to request services from remote computational resources (Java Remote Method Invocation [21]). A generic methodology is available to transfer data over the network in a consistent format (Java Object Serialization [20]). Users are provided the de nition of interfaces to be implemented by plug-ins implementing the basic services. The trade-o between portability and e ciency for the di erent components of the framework could be tuned as appropriate. However, the adoption of Java does not impose any major constraints on HARNESS applications. In fact the only strict requirement is to encapsulate services into a Java class; the adoption of advanced Java technologies such as object serialization and RMI in user developed plug-ins is completely optional. The capability to trade portability for e ciency by using native code inside plug-ins is extremely important { in fact, although portability in general is needed in all components of the framework, it is possible to distinguish three di erent categories among the components that require di erent levels of portability. The rst category is represented by the components implementing DVM status management and load/unload services. We call these components kernel level services. These services require the highest achievable degree of portability, as they are needed to enroll a computational resource into a DVM. The second category is represented by very commonly used services (e.g. a general, network independent, message passing service or a generic event noti cation mechanism). We call these services basic services. Basic services should be generally available, but it is conceivable for some computational resources based on specialized architectures to lack them. The last category is represented by highly architecture speci c services. These services include all those services that are inherently dependent on the speci c characteristics of a computational resource (e.g. a low-level image processing service exploiting a SIMD co-processor, a message passing service exploiting a speci c network interface or any service that need architecture dependent optimization). We call these services specialized services. For this last category portability is a goal to strive for, but it is acceptable if they are available only on small subsets of the available computational resources. These di erent degrees of required portability and e ciency over heterogeneous computational resources can optimally leverage the capability to link together Java byte code and system dependent native code enabled by the Java Native Interface (JNI) [13]. The JNI allows development of those parts of the framework that are most critical to e cient application execution in a low-level language, and also permits the desired level of architecture dependent optimization to be introduced into them at the cost of increased development e ort. 4 The use of native code requires a di erent implementation of a service for each type of heterogeneous computational resource that needs to deliver that service. This fact implies a replicated development e ort for each plug-in incorporating native code. However, if a version of the plug-in for a speci c architecture is available, the Harness metacomputing framework is able to fetch and load it in a user transparent fashion; thus users are screened from the necessity to control the set of architectures their application is currently running on. To achieve this transparency, Harness leverages the capability of the JVM to let users rede ne the mechanism used to retrieve and load both Java bytecode and native shared libraries. In fact, each DVM in the framework is able to search a set of plug-in repositories for the desired library. This set of repositories is dynamically recon gurable at run-time and users can add new repositories at any time. For further details about the features and system architecture of the HARNESS framework we direct the interested reader to [16, 15]. 3 The JavaSpace for Harness Implementation The availability of plug-ins is critical for the attractiveness of the Harness system. The more plug-ins are available, the easier the task for the application programmer will be. Since the Linda [4] model is a well known coordination model for distributed applications, its Java incarnation called JavaSpaces was implemented as a Harness plug-in. 3.1 The JavaSpaces Model and the JavaSpace Interface Sun Microsystems, Inc. de ned the JavaSpaces model and its interface in [22]. The JavaSpaces model is part of the Java distributed computing e ort and plays a central role in the Jini connection technology. JavaSpaces technology is a simple uni ed mechanism for dynamic communication, coordination, and sharing of objects between Java technology-based network resources like clients and servers. JavaSpaces technology is very similar to the Linda [4] model. Instead of holding tuples, a JavaSpace holds entries, which are instances of classes implementing the Entry interface with some additional restrictions1. Templates are used to match entries in the space. The main di erences between Linda and a JavaSpace are (see [22] for further details): JavaSpaces entries are typed and the type is used together with the eld values for matching. JavaSpaces entries may have methods associated with them. The typing of entries allows matching of subtypes, equality is not required if the entry is a subtype of the template. The JavaSpaces technology support transactions which can span multiple spaces. Entries in a space are leased, helping the garbage collector. The JavaSpaces model has no equivalent to Linda's \eval". The JavaSpaces model allows clients to be noti ed when a matching entry is written to the space through remote events. 1Each Entry class must provide a public no-arg constructor. Only public elds of entries are considered. Entries may not have elds of primitive type (int, boolean, etc.), although the objects they refer to may have primitive and non-public elds. 5 The JavaSpaces model uses a di erent nomenclature than Linda: Linda tuple template actual formal out in rd JavaSpaces entry template value wild card write take read The operations allowed on a JavaSpace are de ned by the JavaSpace interface, shown for brevity without the exceptions thrown by the di erent methods: public interface JavaSpace { Lease write(Entry entry, Transaction txn, long lease); Entry read(Entry tmpl, Transaction txn, long timeout); Entry readIfExists(Entry tmpl, Transaction txn, long timeout); Entry take(Entry tmpl, Transaction txn, long timeout); Entry takeIfExists(Entry tmpl, Transaction txn, long timeout); EventRegistration notify(Entry tmpl, Transaction txn, RemoteEventListener listener, long lease, MarshalledObject handback) Entry snapshot(Entry e); } The write() method writes the given entry into the space. The write operation will only succeed if all other operations in the same transaction txn succeed. The lease parameter speci es the initial lifetime of the entry in the space. The lifetime of the entry can be extended through the lease returned by write. When the lease expires, the entry is automatically removed from the space and its memory will eventually be reclaimed by the garbage collector. The read(), readIfExists(), take() and takeIfExists()methods are very similar. They all take a template tmpl that will be used for matching entries in the space. Templates have their elds either lled with values or null. A null eld means \match anything", a eld with a value has to match exactly. Matching exactly means that equals() of the elds marshaled representations has to hold between the template eld and the corresponding entry eld. The txn parameter speci es the transaction under which the method will be executed. And the timeout parameter tells how long the client is willing to wait for a transactionally proper matching entry. The ...ifExists() versions return immediately with a matching entry or null if no such entry exists. The other versions block until such an entry is written to the space. The read...() methods simply return a matching entry from the space, the take...() methods also remove it from the space. The notify() method is used by a client to announce interest in some kind of entry. When entries are written that match the given template tmpl, the space noti es the given listener with a RemoteEvent that includes the given handback object. Matching is done as for read and take. Finally the snapshot() method returns a special instance of an entry that can be used everywhere the original entry could. Using this instance is more e cient than repeatedly using the original entry. A JavaSpace implementation usually stores entries in di erent format internally. snapshot() helps to avoid unnecessary conversion between the external entry form and the internal format. 3.2 Implementation Overview The Implementation is split into four di erent layers, some of them implemented as a Harness plug-in. The Harness plug-ins allow the user to substitute di erent implementation for the 6 layers, depending on the application, available hardware and software. Figure 2 gives an overview of the JavaSpace for Harness implementation. The layers will be discussed in a top down fashion in more detail in the following sections. IStoreInterceptor RingInterceptor put() lookup() remove() removeAsync() handle() RIGroup Ring sendToAll() sendTo() send() sendReceiveSync() sendReceive() receive() Store put(...) process(...) Memory Request Queue JSpace write(...) read(...) readIfExists(...) take(...) takeIfExists(...) notify(...) snapshot(...) JavaSpace H_plugin Figure 2: An overview of the JavaSpace for Harness implementation. The leftmost layer (JSpace) implements the JavaSpace interface. The store layer acts as entry cache. The interceptor layer is responsible for a consistent view of the space among di erent nodes and the group layer implements the ring-based group communication. Group communication is based on other plug-ins for low-level communication. 3.3 Implementation of the JavaSpace Interface and the Store The JavaSpace interface is implemented by the JSpace class. This class mainly passes writes to an instance of the Store class and converts all the read and take calls into requests which are executed by the store's process() method. The main components of the store are the request queue and the memory. The request queue holds outstanding requests from previous process() calls. The memory holds all entries written to the space which were not previously removed by take() or takeIfExists() calls. Therefore the memory represents the state of the space at a given instant. The store's process()method rst tries to satisfy a read or a take request from the memory. If a matching entry is found, it is returned. If no matching entry in the memory is found, null is returned in the...IfExists cases. If no matching entry is found in the other cases, the request is added to the request queue and the calling thread is blocked. The put() method rst scans the request queue if there is an outstanding request which may be satis ed by the new entry. In that case, the request is satis ed, removed from the request queue and the thread waiting on the request is unblocked. If there is an outstanding take request that matches the new entry, it can be satis ed directly and the entry never has to be stored in the memory. In all other cases, the entry is stored in the memory for further use. The main task of the memory is holding the JavaSpace content and providing template matching. In order to provide near-constant matching time for entries, hashing is used for di erent features of entries. Figure 3 shows the data structures involved for matching entries in the memory. The \same or subclass bitmap" provides fast testing if two classes are the same class or if one class is a subclass of another class. This bitmap is updated every time a new (entry-) type is added to the space. If no values are given in a template, matching has to occur based on the type of the entry and the type of the template. For every (entry-) type in the space there exists a list in the \Type lists" data structure. Together with the same or subclass bitmap, the type lists are searched 7 ID hashed hashCode() hashed Type lists Same or subclass bitmap Memory Figure 3: The data structures used by the memory to provide near-constant matching time for entries in the memory. until a type list is found which is the same class or a subclass of the template and contains an entry. Every entry has an ID that unambiguously identi es it. This identi er is used as a hash value in the \ID hashed" table and provides fast lookup of entries based on their identi ers. In the case where values are given for some entry elds, the \hashCode() hashed" table is used. When an entry is written to the space, the hash code of each of its elds is calculated. The entry is then added for each of these hash codes in the corresponding list in the \hashCode() hashed" table. The \hashCode() hashed" list holds therefore entries with elds having the same hash code, independent of the eld position and the type of the entry. Figure 4 illustrates the process of adding an entry to the space. To match a template containing one or more non-null elds, the hash codes of these elds are calculated in the same way as it is done for the entries. These hash codes are then used to nd the shortest list in the \hashCode() hashed" structure. Finally the shortest list is searched for an entry matching the template. anEntry field[0].hashCode() = 34 field[1].hashCode() = 12 field[2].hashCode() = 1 field[3].hashCode() = 4321 ID = 1234 type = EntryType ID hashed size = 8 hashCode() hashed size = 8 Type lists size = 4 Same or subclass bitmap EntryTypeA EntryTypeB EntryTypeC EntryType EntryTypeA EntryTypeB EntryTypeC EntryType E n trTyp e E ntrTypeC E ntrTypeB E ntrTypeA add type insert at (1234 % size = 2) insert at (34 % size = 2) insert at (12 % size = 4) insert at (1 % size = 1) insert at (4321 % size = 1) 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Figure 4: Writing an entry to the space. First, the type is added to the \same or subclass bitmap". Second, the entry is added to its type list. Third, the entry is added to the \ID hashed" table. Fourth, the entry is added to every list holding entries with elds having the same hash code. 8 3.4 The Store Interceptor In order to allow di erent back ends for this JavaSpace implementation, the store supports propagating speci c requests to a store interceptor. The store without a store interceptor is fully functional implementation of a JavaSpace. In such a setting, the memory provides transient storage for the JavaSpace. Using the JavaSpace in that way limits its access to the threads running in the same address space as the JavaSpace. Although it may be made remote accessible through RMI, it does not o er more services than Sun's transient \outrigger" implementation. In order to o er more services like fault tolerance, persistence and so on, the JavaSpace for Harness implementation supports di erent store interceptors. The store interceptor is responsible for a consistent view of the space among the di erent client applications, maybe running in di erent address spaces on di erent hosts. Abstract Distributed Virtual Machine (DVM) A A A ADistributed Virtual Machine (DVM) A A A A JS JS JS JS GCSI GCSI DBSI DBSI DB Figure 5: Two possible implementation of a store interceptor. (A) Application, (JS) JavaSpace for Harness, (GCSI) Group communication based store interceptor, (DBSI) Database based store interceptor, (DB) Database. It is up to the store interceptor how to enforce consistency. The example on the left hand side of in gure 5 uses group communication to synchronize access to the replicated store. The interceptor on the right hand side of gure 5 is a classical client / server implementation with a database as the persistent store. The Harness implementation replicates the space over multiple nodes. Every node runs the same implementation, there is no special master or server node. Every node uses IP-multicast to distribute newly added entries to the other nodes. As IP (UDP/IP) multicast is not reliable, each node might not have a complete copy of the space. However, if a query cannot be completed locally, reliable group communication is used to perform a global query thus consistency is guaranteed. Reliable group communication is also used to atomically remove entries from the space. 3.5 The Group Communication Reliable group communication is used to satisfy to following consistency requirements among the nodes sharing a JavaSpace: Removal of entries from the space have to be atomic. That means that an entry may not be removed more than once from the space. 9 Requests must be seen by all nodes. That means that requests have to be distributed to all nodes upon arrival. The current implementation of JavaSpaces in Harness uses a plugin providing group semantics by means of a ring abstraction. Atomicity of the action performed is guaranteed by the presence of a token that any node needs to hold to initiate an action. The reliability of the ring is guaranteed by the reliability of the links composing the ring together with the underlying Harness system. Each node in the ring is reliably noti ed by the Harness system of any other node failure. When a failure occurs, the node upstream from the failed node queries the Harness system to get the handle of the node downstream from the crashed node and re-establishes the ring. Any node with a pending message (i.e. a message that has not circulated back to it yet) inserts a dummy message in the ring; ordered forwarding performed by the nodes guarantees that any message that is still pending at the time the dummy message is received was lost on the failed node and has to be re-circulated. The number of requests circulating in the ring is reduced by grouping requests together. The links of the ring itself are provided by a Harness plug-in delivering reliable point-to-point communication. Currently the plug-in providing point-to-point communication for our implementation of the JavaSpaces adopts generic TCP connections; however, the modularity provided by the opaque interfaces of Harness plug-ins allows replacement if advanced, high performance communication fabrics (e.g. myrinet, SCI or VIA) are available. Similarly, the plug-in providing reliable and atomic transport by means of the ring abstraction can be replaced by a di erent scheme. In fact, any plug-in providing the Harness atomic-reliable service interface can be adopted. At the moment, only the ring scheme is available; however, in order to overcome the limitations inherent in the ring scheme, e.g. the large number of messages necessary to perform an atomic operation, a di erent plug-in based on reliable multicast [23] is currently being evaluated. It should be noted that due to the Harness opaque interface based plug-in concept, any change introduced by a new plug-in will not necessitate any changes in the upper layers of the JavaSpace implementation. 4 Performance results We focus here on the data transfer behavior of the implementation only, because we are evaluating other group communication protocols at the moment. The goal was to verify the initial assumption we made upon the dynamic behavior 2 of the ring based store interceptor. Obviously, running the implementation without a store interceptor provides the best performance (roughly ten times faster than Sun's \transient-outrigger" implementation but limits its usage to a single address space. Accessing such a space through RMI is still twice as fast as the Sun implementation but has no other advantages beside that. As stated earlier, we were interested in the data transfer behavior of the space. Our test program consisted therefore in one node that wrote data to the space and n nodes reading / taking the data from the space. Data structures were not optimized and the Java just in time compiler (JIT) was disabled. The space was running on a cluster of Sun Ultra 10 workstations interconnected by a 100 MBit Ethernet. Writing to the space is about twice as fast as with the Sun implementation (10521 ms vs. 27568 ms for a 20x20 matrix3), reading is between four and seven times faster than the Sun 2Constant time read() and write() due to multicasting and a linear increase of removal time (take()) with the number of nodes due to ring-based group communication. 3Each matrix element consisted of an ident er (String), its row (Integer), its column (Integer) and its value (Double). 10